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ABSTRACT 



Researchers and program developers in the area of creative problem-solving must 
identify criteria in which complex, real-life problems are involved in order to 
demonstrate the effectiveness of instructional programs which curport to liTiprove 
•creative problem-solving abilities*. T'le f:irst purposes of this project were to 
develop a new measure of Creative Problem-Solving (Improving Research Testing) 
and to develop nev scoring criteria for the College Situations Problems. 

In addition, evidence concerning the reliabilitjr and validity of nine new variables 
derived from these two measures X'jas examined* Adequate levels of , inter -problem 
and irter-rater reliability were obtained for each of the nine variables. Sevan 
of the nine nevj variables were also correlated positively and significantly 
(although moderately) with a variety of external criteria commonly associatad with 
creativity, including divergent thinking abilities and several scales representing 
non-academic accomplishments, 

finally, the nine new variables were applied in comparing four experimental groups 
and a control group in the Creative Studies Frojecto Significant results^ favoria 
experimental groups, \/ere obtained for three of the five variables derived from 
the Improving Research Testing measure, but not for the four variables derivod frci 
the College Situations Problem. Implications for subsequent research were also 
identified « 



CriAPTER I 

I^TTRODUCTIOfl ArD STATE^IB'T? OF PROBLEIi 

In the tvjo dccodcG since Guilford's (1950) discussion of the lack of research 
interest in creativity, activity among educators and psychologists has increasca 
rapidly^ One of the most extensively studied problems is whether or not it is 
possible to nurture creativity through training or instructiowi,-. • Iiaay racthods and 
techniques have been advanced, and several instructional programs and curricula 
have been developed o These purport to have, either as their principal goal or 
at least as a goal of some significance, the facilitation of creativity (Trcffinger 
and Gowan, 1971) d Utilizing many of these nethods and programs^ numercus exper- 
imental tosts have been conducted, particularly among children and adolescents ^ 
The results of many studies sut;gest that it is poss: jle to facilitate many diver- 
gent thinking abilities through deliberate training or instruction,. 

It seerr.3 very important^ hoiv-jever, to inquire whether or not such training attempts 
also have more complex effects However important the divergent thinking abilities 
r.iay be as a necessary aspect of the measurement of creativity (Guilford. 196 ''a, 
1971). the complexity of the creative process indicates that many different abil- 
ities, and some non-aptitude traits, influence in significant ways tho developir.r-jr?t 
and expression of creativity* Although the divergent thinking abilities sre 
important criteria in creativity-training research, they shccld ''not br considered 
sufficient criteria (Treffinger, Renzulli, and Feldhusen^ 197l)o 

E\^en though there is evidence that divergent thinking abilities are correlated 
with creative productivity, it does not necessarily follovj, of coarse^ that 
training which facilitates divergent* thinking will also result in increases xn. 
creative behavior (more generally defined ) o The question is., then. '^1/111 ivicreasin 
divergent thinking also result in increased effectiveness in r.ore complsx orerAx-^/s 
problem-solving?" Of what use are increased divergent-thinkir^g abilities;, wo 
might askp unless they contribute to a person's ability to solva more quickly cr 
effectively the many complex problems which life constantly presents? 

There are also several conceptual and methodological problems relating to the use 
of many a?ci.sting measures of divergent thinking^ which further suggests that 
additional criteria are needed for a comprehensive assessment of the effects of a 
training program.. 

Critical problems for the researcher or program developer^ thon^ are to identify 
criteria which represent the complex problem situations with which thT^subJect 
mst deal in his ordinary life "experiences (i^.e«, "real life" or '^naturalistic" 
problems)^ and to, demonstrate that the training has had a facilitating effect 
on such criteria, as well as on less complex measures (such specific: divergent 
thinking tasks) c This represents^ in one sense^ a step toward the ultimate 
criterion for creativity trainin[; research: does the training infru^nce'"the per- 
&on^s Tehavior in naturalistic problem-solving settings? 

Although it might be most advantageous and enlightening to follow a group of 
trained and control subjects over a "^ong period of time^ as they deal with their 
real problems, such procedures are clearly impractical^ In the proposed project^ 
therefore^ an attempt will be made to develop and validate new criteria which 
will meet the requirements of research utilis^ation, and, at the same time, assess 
outcomes more complex that those assessed by existing measures of creative and 
divergent thinking.. These criteria will then be utilized in the overall evalua- 
tion of the Creative Studies Project^, currently being conducted at the State 
University College at Buffalo, New York (Parnes and Noller, 1973) <> 



Review of Literature 



In this seoviion, tvro important areas of concern in ths proposed research will bs 
reviewed: first, literature pertaining to techniques and programs for facili-- 
tating c.-eative behavior; and second, recent theory and research concerned vjitii 
problems in the measure j.ont of creative thinking and problem solving^ 

Fa cilitation; Techniques au^ Proi^- ramso Wide-*spread interest in creativity in the 
last two decades ha? resulted in the development gi many procedures, methods^ and 
prog?/ams for nurturing creativity^ Treffinger and Gowan (j971) have listed over 
forty different approaches and programs. Among these, many have been developed 
as instructional programs for utilization in school settinr.*Sc Numerous research 
studies have been conducted in which their effectiveness has been tested under a 
variety of conditions, 

Torrance ^nd his associatea (Torrance and Gupta, 196b; Cunnington and Torrance, 
1965 J i'"^-^r and Torrance, 1965) have developed printed and recorded programs for 
fosterir:^ creative thinking among children and early adolescents 1 An instruc- 
tional program for eler.icntary school children, in which several techniques are 
integrated, has been developed and tested by Davis and Houtman (1968) r Covington, 
Crutchfield, and Davies (1966) developed programed instructional materials for 
facilitating creative problem-solving among fifth- and sixth-grade pupils j these 
materials have been utilized in several research investigations (Treffinger and 
Ripple, 1971) « The Purdue Creativity Training Pro gram (Feldhusen^, Treffinger. 
and Bahlke, 1970), is a series of 2li audio-taped^rograms , each with accompanying 
prii^ted exercises, to facilitate creative thinking and problem solving among 
pupils in the upper elementary grades* Evidence for its effectiveness has beon 
preseiiced in several studies, including a large-scale test involving forty-eight 
fourth-, fifth-, and sixth-grade classes (Feldhusen, Treffinger. and Thomas. Ip71)- 
Synectics Educational Systems has also produced a broad range of educational mat- 
erials ^ designed to facilitate creative talent among students at many age levels ; 
from childhood to adult (Gordon, 1971)o 

One of the most extensive programs of research and development,, particularly in 
relation to training programs for high school students, college students and 
adults, has been the program developed at Buffalo by Osborn (1953) and continued 
and expanded^ by Parnes and his associates (1967a )« Their efforts have included 
a wide range of programs and research investigations, including: 

(1) establishment 3nd evaluation of a college-level elective in crea- 
tive problem-solving (Parnes and Meadow 19^9; Meadow and Parnes, 1959); 

(2) a week long Creative Problem-Solving Institute, for adults in 
7ndustry, government, and education* This has recently completed its eighteenth 
annual program ( Journal of Creative Behavio r, Vol. 5, No<, 1971, ppo 28l-290c 

(3) and, an experimental program with high school pupils (Parnes, 1966).. 

There is novj in progress at the State University College at Buffalo, New York, 
a Creative Studies Project, which extends this line of research (Parnes and Noller. 
1973). Incoming freshman students who applied for participation in the Project's 
program were randomly selected and assigned to experimental and control groups e 
Students in the experimental condition participated in a four-semester sequence 
of courses in creative problem-solving r Assessments of their growth in creative 
abilities were made, including comparisions . of experimental and control subjects 
on a wide range of cognitive and affective variables (Parnes and Noller, 1973)o 
Longitudinal comparisons are also planned ^ (Experimental group instruction began 
in September 1970» ) 

Research with many of the programs discussed above leads quite clearly to the 
conclusion that it is possible to effect significaint facilitation of divergent- 
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thinking abilities n Many of the studies cited above , using a variety of training 
procedures and programs, have shown quite consistently tha*** .creative thinking 
abi].itics can be nurtured (Meadow and Parnes, 1959j Britton^ 196? j Torrance, 1965; 
Torrance and Ilyers, 1970j Feldhusen, Treffinger, and Thomas," 1971j Parnes, 1967b; 
Guilford^ 1967 )e There is also some evidence that the effects of training persist 
over time (Covington and Crutchfield, 1965; and Parnes and ileado^^, I960), and th3"i'. 
instructed Ss are superior to controls in performance on several problem-solving 
tasks (Olton, 1969; Ripple and Dacey, 1967; Tref finger and Ripple, 1971) and on 
measures of personality variables and attitudes (Parnes and Meadow, 1959; 
Tref finger and Ripple, 1969 )« There is very little empirical evidence, hovjevsr, 
concerning the effects of training programs or procedures on more complex, nat*- 
uralistic or "real-life" problem-solving criteria. 

Limitations of '-Existing Ileasures , ..Researchers have been concerned with the dev-- 
elopment of valid, reliable, useable measures for creativity for many years (e^g^, 
Guilford, 1950, 1967a; Torrance, 1962^ 1966^; Mednick^ 1962);, and it is clear thnt 
substantial progress has been made in this area (Guilford^ 1967b; 1970) o However., 
as Treffinger, Renzulli and Feldhusen (1971) pointed out, there are many problei^is 
which still remain unsolved. 

The measures of fluency, flexibility, originality, and elaboration v/hich are 
yielded by several existing measures (Guilford, 1967a; Torrance, I966) do not 
provide a comprehensive measure of creative potential. Divergent thinking scores 
appear to represent a necessary, but not sufficient, component of the assessment 
of creativityo Guilford (1967a, 1971) has argued explicitly, for example, that 
(in Structure-of -Intellect terminology) other abilities, including those which 
involve transf ormations as products, and several which involve b ehavioral conbent-^ 
are very likely to be important aspects of creativity* In addition, a . number of 
personality and affective variables are very likely to be important, components cf 
creative potential (Delias and Geier, 1970), 

Covington (in press) described many aspects of the process of creative probD.em- 
3olving, and concluded th^t measures of divergent thinking, or any attempt to. 
assess creativity through customary psychometric '^objective-testing'^ procedures, 
would be likely to be inadequate. He contended that, particularly because of the 
complexity of the creative process, any attempt to assess creativity which relied ' 
exclusively on ^'factorially pure" tests of specific aptitudes would be non- 
comprehensive. Guilford (1971) also argued that no single aptitude, nor even a 
set of measures .,of similar aptitudes, such as divergent thinking abilities, should 
be expected to correlate very highly with other criteria of creative potentials 

Thus, existing measures of creative thinking seem, logically, to be limited in 
'^''alue. Certainly they, constitute one important criterion to be considered in 
creativity training research, but they should not be, the sole criteria upon which 
ube researcher or developer relies. 

There are also technical and methodological limitations of existing measures, 
which bear importantly on our use and interpretation of such measures* Existing 
measures have been criticized and problems identified by Ausubel (1963), Wallach 
and Kogan (1965)> Wallach (1968), Thorndike (1963)> Vernon (I96I4), Harvey (1970), 
Clark and Mirels (1970), Taft and Rossiter (1966), Guilford (1971), Treffinger, 
Renzulli, and Feldhusen (1971), Khatena (1971), Tryk (1968) and Covington (in 
press) o In general, the problems which are most frequently discussed involve: 

(1) Problems of reliability , especially in relation to the test-retest 
reliability of existing measures, which is frequently substantially lov^er than 
desired over short intervals, and is seldom as high as reliability coefficients 
for measures of other cognitive abilities (such as IQ)# 
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(2) Problems of usaability , which have to do with such concerns as the 
effects of variations in time limits^ testing conditions, administration procedures 
and directions, as well as issues concerning the objectivity and accuracy of 
scoring. 

(3) Problems concerning the internal sti'ucture and construct validity 

of existing measures, including those deriving from factor analytic investigations, 
and such issues as the comparability of sub-tests and the combination of scores 
derived from several sub-tests ♦ 

(U) Problems of predictive and concurrent validity , with two particular 
common concerns: (a) that there is insufficient evidence that scores on existing 
measures either predict future creative accomplishments or are significantly cor- 
related with other acceptable concurrent criteria of creative behavior; or 
(b) that scores on such tests commonly intercorrelate better with measures of 
other aptitudes than among themselves or among other measures of similar aptitudes « 

(5) Problems concerning the content validity , or the theoretical and 
conceptual adequacy of tests in relation to what is known about creative potential 
from other sources. . Covington (in press), for example^ has identified several 
ways in which conventional psychometric procedures seem unsuitable for ?':sessing 
creative ability, and hence, several ways in which many existing measures are 
limited; these are summarized in Table 1. 

Table 1 

Discrepancies Between Implications of Knowledge About Creativity 
and Conventional Testing Procedures 
(Based on Covingtor, in press) 



CREATIVITY I TPLIES: 



1. Deep, personal commitment 
and involvement in problem 



1 • Artif icia 1 , highly 
contrived situations 



Complex problems, unique to 
the person 



2. Formalized, simplified, 
standardised tasks 



3. Serious effort; personal 
consequence involved 



3 • Imper s una 1 , f r e quent ly 
frivolous tasks 



U» A single problem considered, 
with intense consideration 
followed by ample time for 
incubation 



li. A large number of 
discrete items, with 
relatively short periods 
of working time 



5# A variety of cognitive and 
affective variilibles interplay 



5. Emphasis on some ^*pure" 

single' aptitude or ability 



6. Emphasis on coordination and 
management of ideacJ, feelings, 
thought processes 



6. Emphasis on products 



?• Initially: a "mess"j followed 
by spontaneous organization and 
manipulation, without external 
direction to do so* 



7i. Clear, concise directions: 
subject knows what to 'do", 
how to proceed, the nature 
of the needed products, etc* 
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The presient state of affairs with respect to the measui^ement of problem-solving 
is also complex. Ferhai.;:? the most conc^ise evaluation of that area was made by- 
Davis (1,966): '^chaotic'^, aany studies have been conducted in which the criterion 
waj a cask constructed especially for the. study; such criteria are seldom used by 
other researchers, and frequently lack evidence concerning validity and reliability 
In addition, attempts are seldom made to analyze the psychological abilities or 
factors assessed in a complex problem-rnolving task. Little attention has bfen 
given to any analysis of the performance called for by the problem-solving t^?i.k' in 
relation to the structure or content of the training program or experimental n.':-.:- 
ipiilaticn. . Yet at the same time , research has .indicated that there does not appear 
to be a sin^ile unitary problem-solving aptitude (Guilford et al^, 1962), Although 
Davis (1966) proposed a dichotomous classification scheme for problem-solving tasks 
there has yet been no empirical verification of the hypothesized dimensions. Final 
ly, since the passing from general practice of introspective analyses of behavior 
during problem solving^ there have been few attempts to distinguish processes which 
are involved in a complex problem-solving sequence, or to assess the subject ^s 
motives, feelings, or reactions durin^ the problem episode. Thus the criterion has 
frequently been reduced to a simple dichotomy (solution/no solution)^ which results 
in the lo^ss of substantial information which might be important in the evaluation 
of an experimental training programs 

Thus, ib is clear that many difficulties Confront the researcher or developer in 
the area of creative training. Existing meas^ores purporting to measure creative 
thinking abilities provide useful psychological information^ and are important oom- 
ponents of the overall evaluation of the effects of training.. But they are also 
limited in a variety of ways. Existing problem-solving tasks are also limited in 
many important ways; there does not appear to exist any single instrument or bat- 
tery of tests for comprehensive assessment of problem-solving abilities© 

Deve lop ment of Mew Measures s Current Research ^ As in any rapidly expanding area 
of research, it is very difficult to ascertain the number of projects now in pro- 
gress in which new measures of creative thinking and problem solving are being 
"developed and tested. However, in the process of developing the Creative Studies 
Project; the Pi^incipal Investigator^ with the assistance of several other re- 
searchers in the area of creativity, conducted an extensive search for, and evalua- 
tion of, criterion measures. In addition, the Creative Education Foundation, 
through its publication, the Journal of C re ativ e Ee h?2vior ., searches widely for 
appropriate information concerning creativity and its measurements As a result 
of these efforts^ several projects have been identified which ^re related to the 
current problem, 

Covington (in press), and his associates at the Berkeley Creativity Project ^ have 
attempted, for example, to develop tests of creative problem-solving in a programed 
instructional format. In these tests, the subject is confronted with a problem 
cir,uation<. As he progresses through the test^ it requires him to generate hypoth-» 
eses about the problem, test them arjainst known and puzzling facts, account for 
discrepancies and new information, and^ eventually^ to describe a final solution • 
Such measures are in an early stage of development , hovrover, and relatively little 
is known about their validity and reliability. In addition, these tests are being 
developed for use with fifth- and sixth-grade pupils, rather than v/ith adolescents 
and adults. They may also be limited by other factors, such as the fact that 
"clues" provided for the subject are frequently rather obvious analogues to the 
solution^ 

Purdue University's "Belmont Project" (Asher et al., 1970) has been working on the 
development of new measures of complex cognitive functioning. This project, how- 
ever, is also in an early stage of development, and, in addition^ f ocusses its ef- 
forts only on the development of measures for accurate assessment of specific abili 
ties among disadvantaged groups of elementary school children. 



Miles (1968) has attempted to develop "real life" criteria for evaluating the 
effectiveness of training in problem solving. However, the measures disveloped 
were highly specific to the training program involved (e-g. creative design stu- 
dents; or Peace Corps Volunteers), and little validity or' reliability information 
has been provided, 

In searching for criteria for use in the Creative Studies Project, over 70 possible 
measures were considereda Althcagh no single criterion measure was identified 
which was determined to be entirely satisfactory in assessing the overall effects 
of the training program, two promising criteria were found (even though each was 
originally designed for a somewhat different purpose )„ 

Goldfried and D»Zurilla (I969) developed a series cT college-related problematic 
situations, in which the subject receives a brief description of a problem that 
might be experienced by any student on a contemporary American university campus 
(i.ea, problems relating to courses, grades, dormitory re'lst ions and rules,; 
relationships with parents and peers, dating, drugs, etc, ), Their purposes were 
to examine subjects^ responses to these cases according to criteria for competence 
or effectiveness of personal behavior, and to assess hox^r effectively students copa 
with such problems. The structure of the test also seems promising, however, as d 
measure of creative problem-solving ability, 

Coelho, Silber, and Hamburg (1962) reported on the development of a projective 
instrument, called the Student TAT, to assess coping behavior among college stu- 
dents • Subjects were presented with pictures representing 10 ambiguous college 
situations. "Their ability effectively to solve specific, potentially stressful 
problems was assessed from their verbal descriptions -of the situation' and their 
description of the behavior of the characters. Again, although the authors^ 
purposes were very specifically to compare various groups on the effectiveness 
with which they cope with problems, there seems to be proBiise for using such an 
instrument as an. assessment of creative problem-solving ability per se. 

Summary ft Although divergent thinking measures represent one kind of appropriate 
criterion for assessing creativity-training, they are limited by several concep- 
tual and technical problems, and should not be considered a sufficient criterion. 
It was therefore postulated that additional criteria are, needed which are valid, 
reliable, useable, based on adequate psychological theory, and constructed so as 
to be appropriate to the content of the training program* In addition, such 
criteria should assess the subject ^s behavior in a situation which is related to 
real-life problems, and which is not perceived by the subject as arbitrary or con- 
trived» -T^jo measures, developed for other purposes, seem to have potential value 
for utilization as criteria for assessing creative problem solving, although 
neither of these, nor any of several other current attempts at developing nevj 
in'=5asures, is entirely adequate in its present form for creativity assessment. 

The Creative Studies Project 

The Creative Studies Project of State University of New York College at Buffalo 
poller and* Parnes, 1973) represents one of the most extensive projects involving 
instruction in creative problem-solving that has ever been conducted. In addition 
to following the performance of experimental subjects, using a variety of tests 
and inventories, through four semesters of instruction, the evaluation of the 
effectiveness of the instruction has been carefully controlled. In addition to 
utilizing a control group which has also been tested over the entire four semester 
instructional period, great effort has been made to continue to study the perfor- 
mance of subjects who withdrew from the experimental program after one, two and 
three semesters. A major focus of the present project, then, was to develop new^ 
complex criteria for assessing the effectiveness of instruction in creative 
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problem-solving among college undergraduate stiidents. 
^pe^if ic Ob.i ectives of the Study 

As part of the general goal of developing, neiri, complex, criteria for measuring 
creative problem-solving abilities, the follot'j'ing specific objectives v;ere 
formulated: 

(1) To develop new scoring criteria in v/hich specific creative abilities 
are assessed^ for problem situations that have already been constructed by 
Goldfried and D^Zurilla (196?); 

(2) To develop a new measure, in which subjects are asked to solve 
"real-life" problems related to their experiences j.n the experimental project; 

(3) To investigate the interrelationships of .the scores obtained from 
the measures described in Objectives One and Tivo. 

(h) To investigate the validity of the newly-developed variables, 
indicated by their relationship with other appropriate external 'criteria of 
creativity« 

(5) To employ the newly-developed variables in the Creative Studies 
Project in order to evaluate their unique contributions to the assessment of the 
effects of the experimental program^ 
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CHAPTER II 
■OHOD 

The purposes of this chapter fire: 

(1) to describe the criteria for developing new measures; 

(2) to describe the ^ev^elopment of new variables for assessing 
creative problem-solving; 

(3) to describe the ievelopment of creative problem-solving scoring 
criteria for the College Situat ons problems; 

(h) to describe the ^.i^ooedures which were employed for investigating 
the validity and reliability of vhe newly-developed variables; and 

(5) to describe the procedures for the^ a^jplication of the ne^^ly- . 
developed indices of creative .problem-solving in the Creative Studies Projecto 

Criteria for Development of Mew Measures 

The first step in developing nex^ measures for assessing creative problem-solving 
in the Creative Studies Project was to establish general criteria for the develop- 
'iTient of measures. Following Miles (1968) and several criticisms of existing meas- 
ures reviewed in Chapter One, the following criteria wer^ ^established : 

(1) Relevance, The measure must sample from a domaj.n of experiences and 
problems familiar to all sub,jects, to enable them to become actively involved in 
the problem as well as to use previous experiences and trainings The performance 
required by the measure should be similar to other performances commonly involved 
in problems encountered by the subjectSo 

(2) Scoring Criteria^ The measure must include at least two general 
scoring dimensions: firsts there must be an ^^effectiveness" dimension (which 
involves the possibility of actually implementing a. solution to the problem pre- 
sented) and secondly^ there must be a "creative^' dimension which may involve one 
or more of the following specific criteria; 

(a) fluency - the ability to enumerate many ideas related to the • 
problem; 

(b) flexibility ^ the ability to "shift" readily among several kinds 
or classes of ideas and solutions; 

(c) originality - the ability to produce unusual or uncommon ideas 
and solutions e 

(3) 7ariety of Solutions, The problems should be "open-ended"^ so that 
many different ideas and solutions can be given. 

(ii) Problem-Solving Time. livery subject should have adequate time to 
tidve the problem(s) presented in the new measur^. The problems must be long 
enough to provide a challenge^ but not so long as to lead to "fatigue" v^hen admin- 
istered as part of a battery of experimental tests e 

(5) Experimental Control of Resources. All materials and resources 
necessary for the development of adequate solutions for -^-he problem should be pre- 
sent and provided by the experimenter in the administration of the new measure. 

(6) . Reliability. Scores derived from new measures should be signifi- 
cantly and positively intercorr elated, and reliable scoring should be possible 
with minimum training and time expenditure, 

(7) Complexity and Reality. The tasks or problems should represent 
moderately complex situations, and consequences of solutions should be real and 
able to be implemented. The problems should not be excessively formalized;, 



simplified^ impersonal or frivolous ♦ 

(8) Number of Tasks. Rafch3r than presenting- the subject v;ith a large 
number 'of discrete items, the problem should be a single problem, or a small set 
of interrelated prpblems. 

(9) Variety of Skills, Traits, and Abilities. The problem should not 
emphasize, through content or directions, any single skxll, ability, or trait 
(such as ideational fluency, listing ideas, or finding "vjrild'^ ideas )• 

(lb) General Directions • The directions lor the problem should not be 
deliberately confusing or misleading, but there should be opportunities for the 
subjects to organize and nianipulrte the .task independently. 

These criteria were followed in developing a new instrument end in developing new 
methods of scoring the uollege Situations Problems* 

Development of A Hew Problem 

One of the first activities of the project was to develop a new measure, which 
was called, "Improving Research Testing" • This measure consisted cf three sep- 
arate sections, presented in immediate succession to all subjectso 

The three sections of "Improving Research Testing" vjere: 

(1) Ideas for improving testing , in which the subject is asked to 
generate as many ideas as possible for improving the overall creativity testing 
operation* 

(2) Making desicions , in which the subject is asked to identify as man7 
factors as possible to use in deciding which of the ideas produced in Part 1 are 
the best. 

(3) Making a plan , in which the subject is to apply his criteria to his 
ideas, and thereby formulate a specific proposal for the improvement of research 
testing. 

Rationale and Scoring Criteria * The Improving Research Testing problem was con- 
structed for use as a criterion measure in the Creative Studies Project for a 
variety of reasons. First of all, it asks initially for divergent production in 
a meaningful context. Although the ability to generate many ideas, to produce 
many different kinds of ideas, or to produce unusual ideas are not unusual ap- 
proaches to the measurement of creativity, tasks usually employed involve situa- 
tions which are not readily familiar in the everyday experience of the subject ^ 

In this measure, however, we feel that an attempt has been made to create a test 
stimulus which is meaningful to every subject (since all have been participants 
in the research testing for the Project). At the same time, however, it is un- 
likely that there will be wide variations in the previous experiences of the 
subjects, a problem frequently encountered in attempts to develop "real-life" 
or "relevant" problem-solving tasks. Thus, subjects are presented with a relevant 
task in which their past experience will have negligible influence on their present 
performance. 

The three stages of the problem provide a measure which can also be shoirm to relate 
to the instructional program which has been used in the Creative Studies Project. 
In Asking subjects first to generate ideas, then to develop evaluation criteria,, 
and finally to produce a specific plan, there is a correspondence with sequential 
creative problem-solving. Since experimental subjects have been taught to defer 
judgment during ideation, to develop specific criteria far judgment when 
evaluation becomes appropriate, and to use their ideas and evaluation 



criteria in a planful, s^/'stematic -way, they should be more facile than uninstruct«>d 
students in it'^eting the demands made by the three phases of this task-^ We would 
expect, therefore, that their ideation will be more productive, their criteria more 
specific and extensive, and their plans more effective and better able to be- im~ 
plemented. 

For Part 1, the purpose of which was to provide an opportunity for subjects to 
produce as many ideas as possible about the improvement of research testing,' it 
was concluded that the most direct and appropriate measure v?as a simple fluency 
score <, 

For Part 2, the purpose of which was to provide an opportunity for the subjects to 
identify .5:;3 many evaluation criteria as possible, ib was again concluded that a 
fluency measure should be employed* Since the purpose of Part 2 was not to ask 
subjects to select any particular criteria or categories^ nor to apply the criteria 
to their previous list of ideas, the proper focus appeared to be the number ox 
criteria produced. 

For Part 3, in v;hich subjects were aslced to formulate a plan, the selection and 
development of scoring procedures was much more difficult o In accord with the 
criteria established for developing the new measures, we sought scores that would 
represent the workability or effectiveness of the subjects' responses, as well as 
a dimension of originality in their responses* Given a problem that x^as very real^ 
with which each of the subjects had had considerable previous experience, the em- 
phasis W£.s cn identifying responses which were not just original, but I'unctional 
as x^eil. Responses to real problems which are only unusual — perhaps to the point 
of being bizarre — vjould not have represented well the goals of instruction in ■ 
creative problem-solving. 

In Part 3, the subjects were asked to use the ideas and the criteria vihich were 
identified in Parts 1 and 2, and thus to develop the best overall plan for improv- 
ing the research testing operation. The purpose of the problem was to determine 
whether or not the subjects could work with their lists of ideas^ including their 
novel or original ones, and with their criteria, to formulate an effective course 
of actiono 

Accordingly, no fluency, flexibility, or originality scores as such were calculated 
for Part 3 5 fluency without- explanation^ or flexibility and/or originality with- 
out implementation, were not sought in Part 3* *We concurred at this point with 
MacKinnon^s (1962) observation that the genuinely creative response must be adap- 
tive to reality. Many possible alternative scoring criteria were examined and 
considered, in an attemptvto develop scoring criteria that would take into account 
factors such as: organization and structure, uniqueness^ possibility for imple- 
r.ienting the plan, and sensitivity to problems and difficulties <, An example of on2 
fcjn^lysis of Part 3, involving five possible scoring dimensions, with seven levels 
ior each dimension, is presented in Appendix Ac Unfortunately, it proved imprac- 
tical to translate those . complex dimensions into scoring criteria which could be 
applied with reliability. Finally,; tberefore^ two basic scoring dimensions were 
developed, for which reliable ratings could be obtained. These were named 
"Workability" and Importance" c 

Wor kability c Ease of applying and probability of success, (1) within the prime 
experimental objective of realistically evaluating growth, and (2) considering the 
major criteria of cost, college and faculty policies, effects on experimental sub- 
jects, arr3 effects on research staff: 

(0) Blank paper^ irrelevant, hostile, states that everything is "fine 
as is" (stated or implied )• 
^ (1) Completely vague or impractical (in terms of criteria and prime 
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objective); may be simply a personal complaint, no spelling out, no sensitivity to 
criteria expressed or implied • 

(2) Detailed paper but completely impractical; or, undetailed but shovys 
awareness to impracticalityo 

(3) Quite impractical as given, serious shortcomings, little or no 
spelling out or sensitivityo 

(I4) Quite impractical even though reasonable amounti of detail and/or 
sensitivity to major shortcomings » 

{$) Somewhat impractical as given, minbr shortcomings exist without 
spelling out how to overcome them or v/ithout showing sensitivity to flavjs (may 
have detail) f 

(6) Somewhat impractical; shows sensitivity to minor shortcomings but 
does nothing about them. 

■ (7) Feasible as explained; no serious shortcoming on criteria; or, most 
apparent shortcomings, if any, reasonably covered by sensitivity and spelling out^ 

(8) Workable explained, 5 or more ideas presented, or-v-more than one 
level of spelling out^ sensitivity to new challenges implied or expressed, 

(9) Very workable plan, $ or more ideas expressed,- or more than one 
level of spelling out, sensitivity e xpressed . (sees problems that could arise and 
solves them). 

Im portanc e^ In relation to accomplishing overall objective of realistically 
evaluating growth: 

(0) Blank paper^, irrelevant, hostile, states that everything is "fine 
as is" (stated or implied). 

(1) Superficial, 1 or 2 trivial ideas, no clear recognition of problem; 
personal complaint. 

(2) No clear recognition of real overall problem. Mostly minor ideas ^ 
Complains for 5elf and others (general complaints). 

(3) Definition of problem implied; shows some awareness of problem 
evolving; gives only minor ideas; shows ^mcre direction. 

{k) Sees. at least 1 important' aspect of problem • — i.e., administration 
of tests, the ^tests themselves, rationale^ /motivation, scheduling, atmosphere.^ . 
people involved. j . • 

i 

A third score yielded by Part 3 of Improving Research Testing was an overall 
rating of the subject's research plan* All subjects^ responses to Part 3 were 
read and rated independently by a counseling psychologist and tv7o researchers, all 
having had extensive experience in creativity research. Each -rater was asked to- 
separate the responses into six categories, fi'om poorest to best in quality; each ■ 
protocol was then assigned a rating from a low , of 1 to a high of 6, according to 
the category-group in which it .had. been classi:Cieda. The overall rating score for 
each protocal was determined by dropping the most extreme of the three individual' 
ratings and computing the mean of the remaining two ratings. 

\ - . . - . ' ' ' • 

Thus, five scores or variables vera derived from the Improving Research Testing 
questions.: . 

(1) Fluency - Part 1 * 

■ (2) Fluency -'Part 2 

(3) Workability - Part 3 ' 
(I4) Importance - Part 3 
(5) Overall Rating - Part 3 

Development of Mew Scoring Criteria 

The* second phase of the project involved. the development of new scoring criteria' 
for. the College Situations Problems (Goldfried and D'Zurilla, 1969). Although 



this instrtunent was developed originally for assessment of psychological adjustment 
among college students, it seemed well-suited for adaptation as an. indicator of. 
creative problem-solying among the subjects in the- Creative Studies Project, The 
problems included were relevant to the life experiences of college subjocts* In 
addition, the problems were open-ended, and could be answered in controlled time . 
periods without special facilities or resources. If reliable scoring criteria for 
creativity could be established, it seemed that the College Situations Problems 
represented a useful -criterion for assessing the effects of instruction in creative 
problem-solving among college students <, 

. ' ... i 

A sample problem follows: " . . M. 

\ 

"As you read the situation we would like you to imagine thi^t you are now in this 
situation. When you have the situation clearly in mind, think of hov; you are most 
likely to react in such a situation. Then in the space below the situation, write 
down your total reaction in specific details" v 

SAMPLE SITUATION: ' . 

'lit is about .a jndnth after the start of classes during your first semester, and 
several important examinations have been scheduled for the same week« The exam- 
ination for your most difficult course has been scheduled for late Wednesday 
afternoon. 

"You are having breakfast on Wednesday morning;, the day of your most difficult 
exam. You feel that you- are inadequately prepared, and your full schedule of 
classes for VJednesday does not allow time for further studying before the exam." 

Four variables were eventually derived. These were identified as: Fluenc y, 
F lexibility J Originality , and Structural Analysis c T^^o additional scoring dimen- 
sions ( Locus of Control and Knowledge and Application of CPS Methods ) were orig- ■ 
inally constructed, but were dropped from the final scoring and analysis because 
' there were so few stable non-ssero scores that no usable information appeared to be 
obtained. Descriptions of each of the scoring, dimensions folloisrs: 

I« piuenov n This score represents' the total number of ide^as that the 
subject produces in response to the problem^ "Ideas" include. all things which the 
person says he will doo Award one point for each idea given^ as long as some spec- 
ific action or behavior is described. This also applies to specific actions which 
are incorporated into more complex responses. That is, an idea should be counted 
every time the student actually describes .something he could doo- Include the re- 
sponse, "do not hing about the problem", if the person- actually sajrs that""this would 
be his course of action in dealing with, the problem. The i^LUENCy score is the total 
nu mber of ideas given . • ' 



II. ^Flexibility !' This score represents the subject's ability to see 
different kinds of possible solutions, # to see alternative ways' of solving the 
problem^ * 

We shall assess this by classifying the kinds of ideas the subject pro- 
duces, using the categories listed- below^ For every idea which is awarded a point ' 
for FUJENCY, one or mora. of these categories will apply. Try to select the category 
which best describes the major part of the idea being rated, althou'gh, if it is 
really clear, you may decide that one idea fits in more than one category. 

So, the person will have category, numbers for -each idea that was awarded 

points for fluency. But we cannot just add up the number of points as we did for 

.fluency, for we are concerned here with the number of different categories . Thus, 

you should go 'o^'er the list of categories for all the- subject *s responses, and 

O jount ap point for each category the first time it appiears. Subsequent uses of the 

IJK^^ame category receive no points. The total' flexibility score is therefore the 
mm ^2^2^ ^ 



number of different categories used by the subjects 

Host errors in scoring flexibility vJill come from two sources: (a) 
missed categories; and (b) missed student responses ♦ To solve these problems^ be 
certain to study carefully the list of categories^ so that you can classify every 
idea quickly and accurately. Read each paper closely, and remember that any ref» 
sponse which has been given a point for fluency must also lit into one or more 
categories « If it fits into a category which the subject has net previously used., 
it gets a point for F lexibility . 

The' Flexibility Scoring Categories are: 

(1) Self -Improvement or Change. A solution in which the primary factor 
is increasing the student's own ability to do something (to think, to feel, or to 
act). The idea clearly involves self -betterment , and usually describes some cog- 
nitive or affective change in the person (I could better myself.... in some way)<, 
This category includes direct mention of incentives to motivate self « 

(2) Peers • Solutions or ideas vjhioh primarily involve peers as the 
means of solution (example: I'd ask my friends to help me with my homework, etc«.<... 
I'd get a new roommate. „ 

(3) Parents or Guardians, as the primary source or factor. 

(h) School Advisors or Counsellors as primary source of help or advice,-. 

(5) Counsellors Outside Schoolo Clergy, medical, legal, psychological 
assistance. . 

(6) Increasing Tangible Resources, Solve the problem by getting more 
money, credit, or through possession (attcinirient ) of new products. (Include get- 
ting a job, getting a loan, etCo) These ' responses involve adding some new re- 
sources isc things to the existing environment, not just better use or modification 
of what's already there » (Better use of self*= category 1^ better use of environ- 
ment or modifications of environment^^ category 8 or 9) (This category was sub- 
sequently Combined v/ith category 

(7) Group Processes, Solutions which principally involve improvements 
in interpersonal relations. (Not just getting a friend 3 emphasis here is on social 
groups rather than individuals).' 

(8) Physical Environment* Changes in the structure of one's physical 
setting or environment (car, dorn, campus^ etCo)-- - but not just adding new things 
(see Category 6)„ Alteration of one's natural, physical surroundings. 

(9) Effective Use of Resources. Solutions vJhich involve the natural 
and ohysical environment as it is, but stress more effective utilization of re- 
sourceSo (Making what I have work more effectively ) a Includes time and sleep. 

(10) Fantasy. Obvious fantasy (make a money treej find a long-lost 
millionaire relative, etc.) 

(11) Redef iniltion. Responses in v/hich the person solves the problem by 
defining it in a different vjay. (viake the "problem" go away by looking at it in 
some new light). (Subsequently combined with category 13.) 

(12) Rest and Relaxation. Solutions in which the principal act is 
avoiding the problem, taking one's mind off it, doing something else, etc. 
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(13) Do Nothing, (Combined vjith Category II), 

(lij) Study Schedule and Pace, Solutions involving establishing a sched- 
ule or pattern for stuay (when? whether to cram?) 

(15) Study Techniques and Aids, Emphasis in the solution on defining 
better ways or inethods for study« 

(16) Emotional Release o Action taken as an integral part of the solu- 
tion. The purpose is some emotional release (erg,, go scream out the window and 
then return to books.) 

(17) Hon-Acaderaic Reward, 

(18) Anticipation of Future Action, 

(19) Cheating. 

(20) Avoida nee of Social Contact or Intsersction. 

III^ Originality ^ Each response in the solution was tabulated, and 
frequencies were established for "key*' actions in each response o Originality x^as 
computed by weighting the frequency distribution, following the procedure described 
by Wilson, Guilford, and Christensen (1962)* 

The distribution of responses and weights is shown in Table 2s 

Tabls 2 

College Situations: frequency Distribution of Responses 



RATI ^ NOo c - ^ RESPONSES FREQULiICgS INCLUDED 

3 138 1 

2 132 2-3-i4 

1 119 - . 5 through 18 

0 108 19 and above 

• 1197 



Thus, there were (l) 136 -responses, each of which were given by only one subjec.t. 
Each such response was assigned a score of three; (2) responses given by two, 
three, or four subjects totaled 132, and each received a score of two; (3) re- 
sponses given by five or more subjects (but not by more thaa 18 subjects) totaled 
119, and each received a score of one] and (Ij) 106 resporses were given by more 
than 18 subjects, and each of these received a zero score. 

After the scoring weights were computed, each subject received an originality 
score; this represented the sum of the weights assigned for individual responses 
included in the subject ^s total solution to the problem, 

IV, Usie of Structural Analysis , This score was derived by reading the 
student ^s entire response to the problem, and then assigning it a rating as 
follows : 

3 points — The student has a very clear, complex plan, and states ex- 
plicitly a sequence of several possible (successive or simultaneous) action^ that 
Would be includ 3d , He then takes each of these ideas, analyzes it, and proposes 
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several possible courses that might be taken, ) 

2. points — The student has a clear idea about what he would do^ which 
may include more than one possible action, but it is not clearly organizsdp his 
ideas are presented for several courses of action, but not in an explicitly stated 
sequence, -^'^ j • 

• 1 point — /The student merely gives one or more ideas which specify par- 
ticular actions he mights-take-; There is no evidence of plan or organization to 
his ideas. 

0 points — - No ideas, or completely irrelevant response c 

The two scoring dimensions which v/ere subsequently dropped from all analyses were 
Locus of Control and Knowledge and Application of CPS ilethodse The Locus of 
Control scores turned out to be almost always a "2"; the Knowledge and Application 
scores could not be derived with satisfactory reliability by scorers without exten- 
sive knov;ledge of the instructional program. The scoring instructions that were- 
attempted were as. follows: 

V. Locus of Control . Read the entire response given by the student. 
Ask whether the student feels that the problem is. one that can be solved by delib- 
erate efforts on his part, or v/hether what happens will merely be decided by 
chance or fate or luck. Award point.1? as follows : 

2 points — Student sees posr^ible solution, and recognizes that he can 
do somethihg about it that will effect a sat.isfactory solution:, 

1 point — Student sees that problem can be solved but. relies- exclusively 
or primarily on others to tell him what to do to solve it* He^s not really in 
commands 

0. point s — Student does not feel that there is any course of action that 
he can take; he feels it is entirely up to others, or to luck, fate, or chance* 

VI. Knoxvledge and' Use of CPS j^ethods ^ Giv^s 1 point for anything the. 
student says which fits into any of the followihg ar6as« (Note that in three cat- 
egories, yoir.may give more than 1 point— 1 for each time the category is found. 
Sot all others, give a maximum of 1 point, per area'<») ' ' 

CPS>iETKODS , ' KiTCvJLEDGE ■ USE 

?ACT-FII1)II]G Student says -he would try Student describes a way to 

to get more information, 'get more information which 

(1 point) he would use. (1 point) 

PROBLEil-FIWDrU Student says he would try -Student actually does offer 

' . — to state a definition of- a definition of the "real 

the problem, (l point) problem", (l point) 

IDEA-?Ir!DIr!G Student says he would try Student actually does gener-* 

■ to get nevj ideas, or that ate a whole list of ideaso 
. ' he would brainstorm. (1 point) (1 poiht) 
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CPS-rlETHODS 



KNWLE.DGE 



USE 



SOLUTIOTT-^ 



riDIXi 



Student says that he would 
develop criteria for judg- 
ing his solution^ or that 



Student acti.ally names the 
criteria or actually uses 
them to evaluate his ideas n 



he would use certain criteria, 
(1 point) 



(1 point ) 



ACCEPTANCE -FI^ D T Jj 



Student says he would try 
to find, ways of making 
certain hi;j solution would 
be accepted and used* 
(1 point ) 



Student actually describes 
details for getting his plan 
accepted o (1 point for each) 



Student names new problems 
which would develop • (1 
point for each) 



Student gives ways of solv- 
ing these new problems ^ too^- 
(1 point for each) 



Pirocedures: In vestig ation of Validity a nd Reliability' 

For each of the five variables derived from the Improving Research Testing measare 
and of the four variables developed for the College Situations Problems^ several 
tests of validity and reliability were conducted.* These included: 



(1) Inter-score correlations 

(2) Inter-rater reliability 

(3) Correlation vjith select^^d cognitive and affective variables related 



(a) measures of divergent production and transformation s ^ from the 
Structure-of -Intellect 'lodel (Guilford, 1967)j 

(b) measures of non-academic accomplishments, using the inventory dev-- 
eloped by Richards, Holland and Lutz (I966)j 

(c) selected items from the Adjective Check List (Gough and Heilbrun, 



of non-academic accomplishments • 

The purpose of these tests were to provide preliminary evidence concerning the 
validity and reliability of the newly-derived measures prior to application of the 
new variables in the Creative Studies Project. 



The sample for this project consisted of 158 subjects who were participants in the 
Creative Studies Project (Parnes and rioller, 1973). 

The total research sample for the project was randomly selected from over 3^0 
applicants for the Creative Studies program at State University College (Buffalo) u 
These applicants represent approximately one third of the total incoming freshmen 
group to vhom the program was offered • from the total body of applicants, l50 
were randomly placed into the experimental group — organized into six class sec- 
tions — to receive the first semester of Creative Studies courser; an equivalent 
number were randomly assigned to the control group (which received no courses in 
Creative Studies until the conclusion of the tx-jo-year project). All testing of 
experimental and control subjects was. done simultaneously, in late afternoons. 



to creativity, including: 




ujgmple 
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evenings, weekends, or other times when no classes were in session for the subjects. 



Instruments 

A variety of instruments-, related to cognitive and affective components of creative 
talent, were administered to all subjects during the Creative Studies Project., 
Scores on these tests provided opportunities to investigate the validity of the- 
^nex-Jly-derived variables. The additional instruments thus employed wsre: 

(1) Divergent Production and Transformations Tests, Ten measures de- 
rived from the Structure-of-Tntellect iodel, representing the divergent producti oii 
operation or the transformation products, were administered, Tfhese instruments 
are considered by auilford (lyb?)^ (Guilford and Hoepfner, 1971) to assess cognitive 
abilities v^hich are positively related to creative talent. Evidence for the valid^* 
ity and reliability for the following tests, which were employed in this project, 
is summarized ard reviewed by Guilford (196?) and Guilford and Hoepfner (1971): 

(a) Alternate Letter Groups (divergent production of figural classes )o 

(b) Multiple Social Problems (divergent production of behavioral 
implications ) • 

(c) Utility Test: Ideational I^^luency (divergent production cf semantic 
units), 

(d) Utility Test: Spontaneous i^lexibility (divergent production of 
semantic classes ),> 

(e) iiultiplo Behavior.'^l Grouping (divergent production of behavioral 
classes )^ 

(f) Varied Emotional Relations (divergent production of behavioral 
relations )o 

(g) Insight Problems (divergent production of figural transformations), 

(h) Verbal-Picture Translations (cognition of semantic transformations)^ 

(i) Homonyms (memory of semantic transformations ) o 

(j) Jumbled l ords (evaluation 6f symbolic transformations ) 

(2) Mon-Academic Accomplishments,, The American College Survey 
(Richards, Holland, and -i'.utz, 1966) is a self-report instrument, which asks the 
subject to report his acoomplishments in each of twelve areas: Leadership, Social 
Participation, Art, Social Service, Science, Business, Kumanisuic-Cultural Activi- 
ties, Religious Service, Music, Writing, Social Science, and Speech-Drama <, 

There is evidence for the inventory's validity (Richards, Holland, and Lutz.. 1966). 
and it seems clear that student accomplishments in the variety of areas included 
by the twelve categories would be an adequate external criterion of creative ex- 
pression among college students (cf., Wallach and i^ing, 1969)o 

(3) The ;.djective Check: Liat« The Adjective Check Li$t (Gough and 
Hv:jilbrun, 1965) consists of 300 adjectives commonly used to describe the attributes 
of a personV These adjectives encompass a wide variety of human traits and be- 
haviors, and 2h separate scales are commonly derived. For the purposes of this 
project, however, attention was focused upon a set of 22 items*. These were sel- 
ected in advance by the instructors of the 'creative problem-solving courses because 
of their particular relevance to creative ability as conceptualized in the instru- 
tional program. The nev/ly-d eve loped measures from this study were correlated 
with the scores derived from this special scale. 

Statist ical Procedures 

After the nine scoring dimensions' had been developed for Improving Research Testing 

and the College Situations Problems, several statistical tests were conducted. For 

all statistical tests, the .05 level of significance was accepted as satisfactory 
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for the rejection of the null hypothesis. 



(1) Inter-j^ater Reliabilities o The first test was to establish satis- 
f^ictory inter-rater reliabilities for all measures. Raters were trained by the 
investigators;^, and were given samples of hypothetical responses until they were 
familiar with' scoring criteria. For each new variaol.e^ a sample of 20 tests was 
rescored, ao that inter-rater reliabilities were obtained for all tests and raters* 
If satisfactory reliability Was unable to be attained (as in the complex matrix 
mentioned earlier for scoring Part III of Improving Research TesJing a^nd for KnovJ- 
ledge and Application of CPS Methods for the College Situations Problems), the 
scoring dimension was dropped from further consideration in the . project. We hope 
that we shall be able to continue revision of these scoring dimensions for possible 
.utilization in future projects. In addition to checking the correlations among 
the raters^ scores, it, was further established that there were no significant dif- 
ferences between the,^~scorers ' means for each scoring dimension* 

^ ■ . ■ . 

(2) Inter-Variable Correlations. The intercorrelations among the five 
variables derived from the .Improving Research Testing measure, and of the four 
variables derived from the-Gol-lege Situations Problems, were also computed and 
tested for significance<» 

(3) Correlations with External Criteria* «Iext, correlations were 
computed between each of the nine newly-developed variah.les and each of the exter- 
nal validity criteria (divergent production and transformation abilities, non- 
academic achievement, and the special scale of selected items from the Adjective 
Check List). These correlations were tested for significance.. 

(h) Comparison of High and Low Criterion Groups on Total NonWicademic 
Achievements The final test of construct validity involved a comparison., on each 
of the nine new variables, of the subjects who scored highest and lowest within the 
entire sample in total number 'of non-academic accomplishments. Subjects v/ho scored 
one standard deviation or more abov e the mean on total non-academic accomplishments 
.(N«20) constituted the, "high" group7 Subjects who scored one standard deviation 
or more below the mean constituted' the 'jlow" group. (N=19). These groups v/ere then 
compared on each of the nine new variables separately, using one-way AMOVA. 

Procedures: Application in Creative Studies Project 

Finally, after the completion of the tests of reliability.. and validity for each 
of the nine new variables, each of the newly— developed variables was utilized as a 
dependent variable for comparing experimental and control groups in the Creative 
Studies Project, This involved a. comparison among five groups: 



(1) those experimental subjects having had four semesters of instruction 

(2) experimental subjects having had three semesters of instruction 



(3) experimental subjects having had two semesters of instruction (N=20), 
(li) experimental subjects having had one semester of instruction (N~31).. 
(5) control subjects^ having received no instruction in creative pro- 
blem-solving (N=57). ' ' ' 

These comparisons, separately for each of the nine new variables, were made using 
one-way A;!OVa, with appropriate post-hoc comparisons • 



Suininary 

In this chapter^ the development of the measure, '^Improving Research Testing" and 
the nex^ scoring criteria for the College Situations Problems were described. Pro- 
cedures for investigating the validity and reliability of nine new variables, and 
for their application in the Creative Studies Project, were also described o 
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CHAPTER III 
RESULTS 



In this chapter, the results of the fjtatistical analy^sss described in Chapter II 
will be presented^, For each of the nine nsw variables, the resul-os will be prs- 
sented for: reliability tests, validation tests, and application in the Creative 
S'iudies Project* 1 

Religbili t V Tes t^ 

Tho reliability of the nine new variables was consid'^red in two ways*. inter-Tstei: 
rel5.ability, and inter -pi' oblem reliability* The Pro^iect testing sohedu'Le did urb 
make possible collection of data for determination of test-retest or alternsite form^ 
reliabilities. 

Inter -Rate r Ro l iability - 

Reliability of raters was checked through applicable correlation coefficients be- 
tW3r;n the scores of in^-lependent rjterSo As indicated by Table 3, the reliabilroioe 
"vjeve uOceptablSj, with only tvc being beloiv the .70'Se 

Table 3 

Inter-Rater Reliability for /ill New Variables 



^^^I'^l^ X SGQPlER 1 X_ SCORER 2 ^C^RELATION 

Research To*stTj-ng * . 



1. 


Flaoncy Part I 


6.7 


6,7 




2. 


Fluency Fort II 




5.6 




3. 


Ivorkability 


h.U 


h,9 




i4. 


Imporiancs 


1.8 


1^8 


„77-);-;s- 


B„ 


ColD.e^e Situations 








1., 


Fluency 




3.3 


.93-^(- 


2. 


Flexibility 


2.7 


2cS 




3. 


Originality . 


h.>7 








Structural Analysis 


1,8 


1-7 


o7l4^Ki- 


Cr 


Overall Ratings : Iraproving Research 


Testing, Part III 





Intercorrelations Among 3 Raters 
Rater 12 3 

1 1»00 „65i«f .5Mh^ 

2 1.00 ..53-;H'r 

3 1,00 



: wfr r yO, p <,01 
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Means and standard deviations for all experiinental variables are presented in 
Appendix B* 



Inter-Problem Reliability 



The correlations among the five variables derived fi-om Improving Renoarch Testing 
and among the four scores derived from the College Situations Problem are presented 
in Table as well as the correlations betvreen both sets of variables. 

Table h ' 
Intercorralations Among New Variablee"!- 



Jjnproving Research Testing 



(1) 


(2) 


. (3) 


(ll) 


(5) 


(6) . 


(7) 


(8) 


(9) 


Flu-I 


Flu-II 


Work. 


Impt 0 


Rating 


•Flu, 


FlaKc 


Origc 


Struct. 


(1) 100 


70-"^i- 


13 


08 


• h3^^ 


26-5Hi- 


00 


19-> 


15 


(?.) 


100 




20^- 




2U«-is- 


09 


10 


15 


(3) 




100 


25^h;- 


37-!^!:- 


01 


-01 


-06 


01 


ih) 






100 




02 


08 


02 


oU 


(5) • 








100 


c? 


. 01 


00 


0? 


(6) 










100 








(7) 












100 






(8) 














100 


25-ii-* 


(9) 
















100 



Ccllege Situations 



1<» /ill decimal points emitted. 

The Fluency score from Part I of* Improving Research Testing was significantly and 
positively correlated with the Fluency Score from Part 11^ with the Overall Ratirg 
from Part III, and with the Fluency and Originality scores for the College Situ.^- 
tion Problem* 



The Fluency Score from Part II of Improving Research Testing v/as significantly and 
positively correlated v/ith: Fluency Part I, Importance, and Overall Rating for 
Part III of Improving Research Testing, and with the College Situations Problem 
Fluency score. 

The Improving Research Testing Viorkability score was significantly and positively 
correlated only with the Importance Score and the Overall Rating.. 

The Improving Research Testing Importance score was correlated positively and sig- 
nificantly with each of the other Improving Research Testing scores except the 
Fluency Score for Part 1^ 

The Fluency, Flexibility, Originality and Structural Analysis scores for the 
College Situations Problem were all positively and significantly intercorrelatedo 



Correlations with External Criteria 

Summarized in Table 5 are the correlations of each of the nine new variable/, with 
each of 23 external criteria (including 10 selected measures of divergent yro- 
duction or transformation abilities from Guilford »s Structure-of -Intellect ^ 12 
scales representing non-academic accomplishments in a variety of areas, and a sCale 



representing selected items from the Adjective Check List). 
Improving Re search Testi ng 

The Fluency scores from Part I of the Improving Research Testing measure were 
significantly and positively correlated v/ith five divergent -'pr eduction abilities^ 
xjith -ohreej non-academic achievement scales (Social Participation^ Humanistic^ 
Cultural^ and v/riting)/ and with the selected scale of Adjective Check List itemv*?o 
The Fluency scores from Part II were significantly and positively correlated vith 
four of the divergent production abilities and with three non-acaderiiic accomplish- 
ment scales (Business, Humanistic/Cultural, and writing). The Workability score 
Was correlated significantly only with the Social Participation accomplishments 
scale, and negatively for that scale. The Importance score was positively and sig- 
nificantly correlated with only one measure (cognition of semantic transformations 
The Overall Rating vjas correlated significantly and positively v/ith four divergent 
production abilities and one transformation abilityo It was also significantly 
negatively correlated with the Artistic accomplishments scale« 



(Continued after Table 5) 



Table 5 



Correlations of Nine Newly-Developed Variables 
VJith 23 Selected External Creativity Criteria 

New Variable"^ 



Criterion 



Am Structure of Intellect 2 



Improving Research Testing \ C ollege Situations 

Flu -I Flu'-II Work- Imptce Rating Flu^Flex Orig Struct 
ability 



Alt. Letter Groups (DFC) 
Mult. Social Prob. (DBI) 
Utilities: Fluency (DIiU) 
Utilities: Flex. (DMC) 
Mult. Behav. Group, (DEC) 
Varied Lmot. Rel. (DBR) 
Insight Probs. (DFT) 
Verbal-Pict. Traasl. (CHT) 
Komonyms (rfl'IT) 
Jumbled VJords (EST) 



09 


U4 ' 


03 


^03 






15 


12 






13 


"02 






06 


-01 




IS 


-03 


-12 




33-!H!- 


GO 


08 




02 


-05 


00 


12 


15 


03 


21iHt- 


-06 


00 




-01 


-02 


01 


11 


Ok 



00 • ??*•«• 
28-4- 22-«-«- 
23-5H^ 29'^ 12 
16^\ 21-tHS- 10 
Ih 
03 
02 
02 
09 
05 



09 
17 

00 ! 



17-H- 

Ih 

02 



00 I 07 
I8^i-i 09 



21^ 16^ 
12 19^ 
20* 
17rc 

09 
01 

-05 
-03 
19^1- 
03 



111 

08 
.06 
07 
05 
00 
00 
09 
06 
07 



B. Hon-Acadcmic Accomplishmenta 



Leadership 


11 


11 


-02 


02 


07 


; 12 


06 


■ 05 


25 


Social Participation 


17* 


09 


-19* 


-06 


00 


19* 


02 


13 




Artistic 


00 


03 


-11 


-li4 


-17* 


08 


00 


-03 


-05 


Social Service 


01 


07 


-111 


13 


-01 


20 


Ok 


11 


23^H«- 


Scientific 


06 


01 


-05 


05 


. Oh 


01- 


-03 


05 


08 


Tiusiness 


12 


16* 


01 


-09 


Olt 


09 


-10 


ck 


08 


Humanistic/Cultural 


29^^ 


22^Hi- 


-Ik 


00 




13 


00 


13 


15 


Religious 


06 


13 


-07 


-13 


-10 


08 


11 


05 


09 


ilusical 


-07 


01 


08 


-07 


-Ok 


02 


00 


-05 


-05 


Writing 


21^Ht 


19* 


-07 


-03 


Ok 


13 


07 


12 


18i^ 


Social Science 


15 


05 


-07 


-.05 


-02 


07 


-06 


12 


16* 


Speech and Drama 


03 


-09 


-03 


-06 


-03 


-08 


-Oil 


Oh 


-06 


C. Selected ACL Items 


16* 


13 


-16 


-03 


-ok 


-01 


-10 


01 


-01 



1. All decimal points omitted 

2. Structure-of "Intellect classification in parentheses* 



College Situations Problems 



For this measure, the Fluency scores were significantly and positively correlated 
with five divergent production abilities, and with the Social Participation and 
Social Service accomplishments scales. The Flexibility scores were significantly 
and positively correlated only with a measure of divergent production of figurai 
classes. The Originality scores were significantly and positively correlated with 
four divergent production abilities and with memory of semantic transformations • 
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The Structural Analysis scores were significantly and positively correlated with 
four non-academic accomplishment scales (Social Participation, Social Service, 
Writing, and Social Science )• . _ 

Summary 

Although each of the nine nev variables was significantly correlated with at- least 
one of the external' creativity criteria, only one of those variables was correlated 
with a broad range of cognitive, personality, and behavioral criteria (Improving 
Research Testing-Fluency, Part I). Very few of the new variables seemed to be 
significantly correlated it^ith transformation abilities, accomplishments in the arts 
or the selected set of items from the Adjective Check List. In addition, even when 
correlations v/ere statistically significant, all were of low to moderate magnitude « 

Total rlon-Academic Accomplishments 

The final criterion for investigating the construct validity of the nine new Vari- 
ables involved the total reported non-academic acc©mpliaftments of tha subjects.. 
Based on the total number of accomplishments ^reported xn all twelve nori-academic 
scales, high and l6w groups'' were identified ^ The 20 subjects In the high group 
(with 20 or more reported non^-academic accomplishments ) 'and the .19 subjects in the 
low group (with four or fewer reported non-academic accomplisbuents)were then com- 
pared, using one-Way AIJOVA, on each of the nine new variables* The results of these 
comparisons are summarized in Table 6^ 

The high and low groups did not differ significantly on four of the five variables 
derived from the Improving Research Testing measuren For the Workability variable, 
the mean for subjects in the low group iU*12) was significantly greaher fhan the 
mean for subjects in the high group (2,C0)^ 

For the College Situations Problem, subjects in the highj non-academic accomplish- 
ments group had significantly greater means than subjects in the low group on 
three of the four new variables: Fluency (i4.^60 vs. 3.00|), Originality (6370 vso 
ii.OO) and Structural Analysis (2.25 vs. 1.72). \ 



(Continued after Table 6) 



Table 6 



One-Way A?IOVA Comparisons of High and Low Criterion Groups on 
The American College Survey (Total Accomplishments) for New Variables. 



Criterion 


High Group (N=-20}-'" 


2 

Low Group (N^^l?) 






Variable 


X 


S.D, 


X 




F(p) 




Improving Res. Testing 












Fluency - P^rt I 


8.9S ~ 


5,77 


5.68 


I1.I2 


3.W 


(n.s, ) 


?luency - Part II 


7.10 


. I4.16 




2a95 


1.95 


(n.s . ) 


Workability 


2.80 


2.39 




2,82 


5.lij 


(:.05) 


Importance 


1.95 


0.75 


2.16 


0,70 


< 1 


(n.s-. ) 


Combined Ratings 


2.65 


1.59 


2.63 


1«52 


<.l. 


(n.s, ) 


College Situations 














Problem 














Fluency 


I4.6O 


1.72 


3.00 ■ 


1.28 


lbo29 


(a .01) 


Flexibility . . 


3.10 


1,07 


2,55 


0.85 


2.95 


(n.Sn ) 


Originality 


6.70 


2,99 


ijcOO 


2.ljO 


9.27 


(<.01) 


Structural Analysis 


2.2$ 


0.55 


• 1.72 


0.66 


7.11 


(<..02) 




lo Accomplishments checked ; 20 (J, S»D. or more above the mean) 
2. Accomplishments checked <^ U (l S.D» or more below the mean) 
Application in the Creative Studies Project - 

Each of the nine newly-developed variables was utilized as a criterion for com- 
paring experimental and control groups in the' Creative Studies Project. There were 
five comparison groups: experimental groups having had four, three, two, and one 
semester (s) of instruction respectively, and the controls, who had not participated 
in any instruction in creative problem'-solving* For each analysis, these five com- 
parison groups are identified as groups one throughvfive, in the order named in the 
previous sentence. 

For each analysis, the sample sizes were: 

Group 1 (four sBmester experimentals ) - 33 

Group 2 (three semester experimentals) - 17 

Group' 3 (two semester experimentals) - 20 

Group i| (one semester experimentals) - 31 

Group S (controls) -57 

Improving Research Testing: Fluency, Part I 

For the Part I Fluency variable, there was a significant difference (p.^j;rOl) among 
the five groups. Post-hoc comparisons using the IJewman-Keuls procedure (Winer, 
1971) revealed that the I4 semester experimental group's mean was significantly 
greater than all other groups' means, and that all experimental group means were 
significantly greater than the control mean. Table 7 reports the means and standard 
deviations for each group and summarizes the AIIOVA for the five groups. 
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Table 7 



Ifeans, Standard Deviations, and Ai^OVA: 
Experimental-Control Comparison of Fluency, Part I 



Means ard Stardard Deviations 



Group 



Mean 10,70 8.6$ 8.I4O ^.hd k.72 

Standard I4.60 ■. 5.37- ^'^^ 3.23 2.72 

Deviation 



i.novA 



Source 



S.S. 



df 



Between Groups 90I4.O9 h 
Within Groups 2210.90 l53 
Total 31lIiT^ Wi 



M.S. F • P 

226.02 l5o6l4 ^.01 



Improving Research Testing: Fl u ency, Part II 

For the Part II Fluency variable^ there was a significant difference (p :>0l) 
among the five groups. Postrhoc cojnparisons using the ATewman-Keuls pro^iedure 
(Winer, 197.1) revealed that all , experimental group means were significantly 
greater than the control mean, that the Li-semester experimental group mean v/as 
significantly greater than all other group means, and that the two-semester 
group^s mean was significantly grefjter than the one-semester group^s mean. 
Table 3 reports the means and standard deviations for all five groups and sum- 
marizes the A'TOVA comparison. 

Table 8 

Means, Standard Deviations, and AiIOVA: 
Experimental-Control Comparison of Fluency^ P<3rt II 



neans and Standard Deviations 

Group 1 1 1 k 1 

Mean 8.70 6.29 1 .00 $.29 3.86 

Standard 2.56 3o53 3.?< 2.l6 l.ll 

Deviation 



Source S.S. df ^I.S. F P 

Between Groups 532*5^4 I 133.114 18,52 . .01 

V/ithin Groups 1099 p 76 153 7.19 ' 

Total 1632.30 157 
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Improving Research Testing: VJorkability 
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^'or the Workability scores, there were no significant differences among the five 
groups. Table 9 reports the means and standard deviations for each group, and 
suiniiTarizes the iiNOVA for the five groups. 

Table 9 

Means, .Standard Deviations, and AilOVA: 
Experimental-Co'itrol Coniparison on V/orkability 



Meaas and Standard Deviations 

Group 1 2 3 U 5 

Mean Ii,l8 2c9h 3.20 3.10- ' 3.81- 

Standard Deviation 2.52 2c$6 3;>0h 2.77 2o56 

A MOV A 

Source S.S.> df . ^I^. F P 

Between Groups 31.9$l5 k 7o9879 ' 1,13 noS. 

within Groups * 1080,6370 ' 1^3 7 30630 

Total 1112. 157 

Improving Research Testing: Importance 

For the Importance variable, there were no significant differences among the five 
groups* Table 10 reports the means and standard deviations for each group and 
summarizes the AilOVA for the five groups. 

Table 10 

Heans, Standard Deviations, and ANOVA : 
Experimental-Control Comparison on Importance 



Heans and Standard Deviations 



Group 


1 


2 


3 


h 


i 


Mean . 


2.21 


1.76 


lc8$ 


2.03 


1.9$ 


Standard Deviation 


0.6$ 


Oo66 


0.7$ 


0.7$ 


0„$8 


AI'TOVA 












Source 


S.S. 


df 


M.S. 


F 


P 


Between Groups 


3.0k09 


k 


0.7602 


1.7ii 


n.s . 


Within Groups 


66.9338 


153 


0.ii37$ 






Total 


69.97h7 


157 
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Improving Research Testing: Overall Rf?tinfe 



For the Overall Rating score, the four ser.iester experimentrl group^s mean was sig- 
nificantly greater than the other four means-o The three and two semester groups 
did not differ significantly frori each other ^ but they were each significantly 
greater than the mean for either the one semester group o'r the controls. The one 
semester group did not differ significantly from the Control group. Table 11 re- 
ports the means and standard deviations for each group and sumariaes the ;*HOVA for 
the five groups.. . 

Table 11 

'.leans. Standard Deviations and i.MOVi» : 
Experimented 1-Control Comparison on Overall Rating 



.leans and Standard Deviations 



Group . * 1 12 k I. 

.lean . . 3k.$$ . 26,l8^ 26,75 2h.3(> 2li,56 

Standard Deviation 10^63 I8067 l6ol7 lii«36 12.^9 

Source . S.S. df f^.S, F P 

Between Groups 2lil|6.95 h 611.738 3.20 >05 

within Groups 292 09>53 1^3 190.912 

Total' 3l6$6.li8 1^7 

Colleg e Situations Problem; Fluency 

For the College Situations problem. Fluency variable, there were, no significant 
differences among the. five groups. Table 12 reports the mean and standard devia* 
tidn for each group and sumrjiarizes the AHOVA for tha five groups.. 

Table 12 

Means, Standard Deviations, and A-p^A: 
Experimental-Control Comparison on College Situations Problem: Fluency 

Means and Stan dar d Deviations 

Group 1 2 3 h 1 

Mean ii.09 3.i47 3.85 ^ 3*32 3*23 

Standard Deviation 1.99 1.28 2,I|3 lo5l 1.86 

ANOVA 

Source S.S. df ^S. ^ F P 

Between Groups 19.0199 h ii.755 1.377 n.s, 

idthin Groups 528,3218 153 3.1i53 



Total 51j7.3iil7 157 
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Collej^e Sittiatiobs Problem: i^lexibility 



7or the College Situations Problem^ i^lexibility variable, there were no significant 
dixferences among the five groups. Table 13 reports the mean and standard devia- 
tion for each group and summarizes the A*107i; for the five groups. 

Table 13 - - • 

'leans. Standard Deviations, and A ^^0'/A : 
Experimental-Control Comparison on College Situations Problem: Flexibility 



ileans and Standard Deviations 



Group 


1 


2 


3 




5 


f iea n 


2,76 


2.76 


2.50 


2.68 


2.65 


Standard Deviation 


1.32 


1.09 


0.95 


0.,98 


i.ia 














Source 


S.S. 


df 




I? 


p 


Between Groups 


1.01 


Ii 


0,2525 


•,1 


n.s. 


ivithin Groups 


193.67 


153 


1„2672 






Total 


19iu88 


.157 








College Situations 


I^oblem: i 


Driginality 









For the College Situations Problem, Originality variable,, there were no significant 
differences among the five groups. Table ih reports the mean and standard devia- 
tion for each group and sumruarizes the AiWA for the five groups « 

Table Ih' 

Weans, Standard Deviations, and A "^bVA • 
Experimental-Control Comparison on College Situations Problem: Ori5.inality 



Means ard Standard Deviations 



Group 


1 


1 


3 


ii 


5 


He a n 




5oio 


il.lO 


14.68 


14.75 


Standard Deviation 


3.83 




2.67 


3.hh 


2.98 














Source 


S.S. 


df 


'US, 


F 


P 


Between Groups 


25.0687 


h 


6,2672 




n.s . 


Ivithin Groups 


1595.3172 


153 


10.lj269 






Total 


1620.3859 
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College Situations Problemt Structural Analysis 



.?or the Structural Analysis variable, there were no significant differences among 
the five groups. Table 1$ reports the mean and standard deviation for each of the 
five groups and summarizes the A'WA for the five groups. 

Table 1$ 

Means, Standard Deviations, and iiiI07A: 
Experimental-Control Comparison on College Situations Problem: Structural Analysis 



Heans and Standard D e viatio ns 

Group i • 1 • 2 k . t 

Mean ■ 2.1$ 1,9k . 1,70 1.81 1.86 

Standard Deviation - 0,$7 ' 0.66 0„73 O.^h 0,69 

A'TO'/A 

Source S.S. df . 'f.S. F P 

Between Groups . 3.2802 h 0o8201 1,99 n.s. 

within Groups 63 o 0995 1|3- 0.kl2k 

Total 66.3797 157 
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DISCUSSITT A B CO-.ICLIISIONS 
The specific objectives which v;ere formulfjted for this project were: 

(1) To develop new scoring criteria in which .specific creative abilities 
are assessed^ for problem situations that have already been constructed by Goldfried 
and D»Zurilla (1969); 

(2) To develop a new measure, in which subjects are asked to solve "real- 
life" problems related to their experiences in the experimental project; 

(3) To investigate the interrelationships of the scores obtained from 
the measures described in Objectives One and Two; 

(h) To investigate the validity of the newly-developed variables, in- 
dicated by their relationship v/ith other appropriate external criteria of creativity 

(5) To employ the newly-developed variables in the Creative Studies 
Project in order to evaluate their unique contributions to the assessment of the 
effects of the experimental program.* 

In this Chapter, we shall suminarize the findin^-S of the project in relation to those 
specific t5bjectives. 

Objectiv e One 

Objective one called for the development of new scoring criteria for the College 
Situations Problei;is^ i^our new scoring criteria were successfully developed for 
College Situations Problems* The new variables assessed fluency (the number of 
ideas produced in response to the problem)^ flexibility (the variety or kinds of 
ideas produces), originality (the ability to produce unusual or infrequent ideas 
about the problem) and structural analysis (involving the or^^cniza tion and 
sequence of ideas produced )« 'Two additional variables x^ere developed, but were 
subsequently dropped because of problems in attaining stable, objective scores « 

Objective Tw.j 

Objective t\<io involves the development of a new, realistic but complex measure of 
creative problem-solving abilities. The Improving Research Testing measure was 
developed to meet this objective. In this measure, subjects x^ere asked to eraploy 
creative problem-solving skills to produce a plan for improving testing procedures 
for creativity reaearch^ The problem was presented in three parts vrhich stressed, 
respectively, ideation (listing ideas for possible improvements), development of 
evaluatio n c rite ria (for assessing the quality or applicability of the ideas listed 
in part one), and, in the most complex section, creation of a systematic plan for 
improving the testing procedures. Frora these three stages of the problem, five nex^ 
scores or variables were developed, assessing (l) fluency (numter of ideas), 
(2) fluency of producing criteria for evaluating ideas, (3) workability (potential 
for actual implementation), (I4) iraportance (value of the proposed plan and changes), 
and (5) an overall rating of the plcn by three expert judges. 

Objective Three 

Objective three concerned the interrelationships among the> nei>fly-developed variables 
The evidence suggested that the nine new variables could be scored reliably by 
college students vath brief training. In addition, the data indj.cated that the 
five variables from the Improving ilesearch Testing measure were positively and sig- 
. nificantly intercorrelated. The four variables from the College Situations Problem 
were also significantly and positively intercorrelated. In general, with only the 
Q exception of the fluency measures, the five Improving Research Testing variables 
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V7ere significantly intercorrelated among themselves, as were all four College 
Situations Problem variables, while correlations between those two sets cf variables 
were negligible^ That is, with the logical exception of fluency scores, variables 
derived from one measure vjere more like each other than they were similar to the 
variables derived from the other measure, we concluded that this was desjrable-- 
since we intended that the new measures vjould yield sorie. hat different information 
about creative problem-solving^ skills and abilities. 

Objective 7our 

Objective four concerned investigations of the validity of the nex/ly-developed 
variables, indicated by relationships viith selected external criteria of creativity.. 
In general;, the results indicated that there vss some support for the validity of 
many of the nine new variables. The iTluency scjres for parts I and II of Improving 
^.esearch Testing and for the College Situations Problem vjere positively and signif- 
icantly correlated with several divergent production measures, a;id with some areas 
of non-academic accomplishracnt (particularly areas involving social activities and 
humanities )• In addition, those subjc-cte who listed the most non-academic accom- 
plxshments also tend.: to have the highest fluency scores on the College Situations 
-problem. 

The Overall Hating for the Improving Research Testing plan was also significantly 
and positively correlated xjith several divergent -production criteria, with one 
transformation criterioh — evaluation of s;\nnbolic transformations — and x^as sig- 
nificantly negatively correlated with artistic accomplisliraents * The Originality 
score for the College Situations Problem w?s significantly and positively related 
to many divergent prediction variables^, to one transformation criterion — memory 
of semantic transformations and to total non-academic accomplishments • The 
College Situations Problein, Structura,l i^nalysis score was significantly and pos- 
itively related to several non-acad<?iiiic accomplishment areas, including social 
participation and service, social scien5<5-^^and^writing« 

The least support i^as obtained for the i orkability and Importance scores for 
Improving Research Testing, each of which was significantly correlated with only 
one of the 23 external criteria, and the ^^lexibilitj^ score for the College Situa- 
tions Problem, which was significantly correlated vjith only one divergent production 
measure* In addition, subjects who clairaed the fewest (total) non-academic accom- 
plishments attained significantly greater Workability scores than subjects with the 
hij;.hest total nui^ber of accomplishments, .Mthough . significant negative relation- 
ships may have as much predictive power as positive relationships of corresponding 
niagnitude, we were unable to recognize any clear logical explanation for 1;he nega- 
tive relationship; the result is puzzling and thescores should certainly be 
regarded very cautiously^ 

It must also be note.d that, althou-^h many correlation coefficients were positj.ve 
and significantly greater than zero, the magnitude of the relationships tended to 
be '»low to moderate." In very feij cases did the correlations exceed .30, 'Thus, 
Vie Conclude that our results provide preliminary indications supporting the 
validity of the new variables, although more .extensive validation is required, 
involving additional and more complex external criteria. The divergent production 
measures, for example, represented only partially the tc^tal divergent production 
categorj'', end did not sample adequately the full range of contents and products. 
In additi' n, Guilford (1971) has warned that correlations of complex criteria with 
only one (or successively with only a few) divergent production categories may* 
yield only moderate correlations. The use of larger test batteries and samples,, 
which would make possible more effective multivariate analyses, must also be 
planned in validating complex criteria (cfi, Treffinger and Poggio, 1972 )« For the 
non-academic accomplishment scales, it should also be noted that the scores on each 
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scale tended to have rather restricted ranges^ which may have caused correlations 
with the new criteria to be somewhat lower than might be expected if a vzider range 
of accomplishment scores had been attained «» In general; then^ we conclude that 
ihe evidence for the validity of the nine new variables is probably somewhat con- 
servative, and is certainly encouraging* 

Objective Five 

The fifth objective was concerned with the application of the nine new vai-^iables in 
the Creative Studies Project (Parnes and Noller, 1973)- i^or each' of the nine 
variables J then^ the four experimental groups and the control group x^ei« compared. 

Significant differences were obtained for the comparisons employing three of thz- 
nine new variables: Improving Research Testing, Fluency, Parts I and II; and the 
Overall Rating for the Improving Research Testing plan> Among the new variables, 
these appeared to be stable, and were supported most broadly in the validation in- 
vestigations <» It was determined that the best performance on each of these three 
measures was attained by the four-semester experimental g,roup^ whose scores were 
significantly greater than the scores of both the controls and the other experi- 
mental groups. All experimental groups t mean scores (except for the one'-yi'ir-gster 
group for the Overall Rating variable) were significantly greater than the cch'trols 
mean scores on all three of these variable§« 

Thus, there was evidence for the sensitivity of the new measures in identifying 
differences among the experimental and control groups in the Creative Studies 
Project. Despite the complexity of the problems presented^ and the development of 
entir^ new scoring procedures, the results of these analyses were remarkably con- 
sistent with the general pattern of the results of the project (cf „ Parnes and 
Noller, 1973) o These measures provided further support, then, for the value and 
cumulative effects of - instruction in creative problem-solving among college under- 
igraduates.-> 

The direction of the actual scores,, even for analyses in which no significant 
differences were found, appeared also to be consistent with the general findings 
for the project to date* Vihile tentative, of course, this suggests that continued 
work on the further development and validation of such criteria, especially includ- 
ing scoring criteria for the College Situations Problems, may be very valuable. In 
particular^ it would probably be quite valuable to utilize the College Situations 
Problems scoring criteria across a more extensive set of problem situations, since 
the scores derived for this study (because of "oime and scoring budget limitations 
i^ere restricted to responses to only one problem situation^ 

Directions for Future Research 

Although we have concluded from this project that useful, complex measures can be 
developed, from which several' valid and reliable variables can be derived, several 
possible dimensions for future research can be identified. 

First, multivariate studies of the validity ';;ind structure of the new variables 
should be undertaken. Although such procedures require greater sample sizes than 
were available for this project (which also increases other practical problems, 
such as test administration and scoring), their utilization would certainly con- 
tribute to our understanding of the validity and reliability of the new variables. 

Next, replications would be valuable, in order to determine whether the relation- 
ships identified in this study do in fact represent accurate estimates of the 
population's parameters. Long-term studies of the predictive validity of the new 
variables, and more extensive concurrent validation against non-cognitive 



(personality) criteria x/ould also be valuable. 

i^inally, further development and research might profitably be addressed to the 
revision and implementation of the scoring techniques that were dropped fror.i the 
present study. 

S um nary a nd C onclus iona • 

The purposes of this project were to develop new variables for assessing the 
effectiveness of instruction in creative problem-solving, to investigate the 
reliability and validity of such variables, and to Uuilize the variables in the 
evaluation of the Creative Studies Project, 

It is concluded that: 

(1) Complex problem situations can yield reliable and valid indices of 
creative problem-solving abilities; 

(2) Application of such indices in the evaluation of the Creative Studi 
Project supports the effectiveness of the instructions! program (especially among 
subjects who participate for four semesters); and 

(3) Additional research, involving multivariate statistical procedures 
and long-term studies, is necessary in order to provide more comprehensive assc'ss- 
ments of the validity of any criteria which purport t'o measure creative problem- 
solving abilities • ' " ' 
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Appendix B 

Means and Standard Deviations for all Project Variables (N'^l^G) 



Variable X S.D. 
Structure-of-Intellect Tests 

Alternate Letter Groups *: 6 ,$3 2 .08 

•iultiple Social Problems. 9^i42 .-2 •70 

Insight Problems 1^6h lo06 

Utility: x^luency ' 17,88 ho 8$ 

Utility: ^exibility 12o30 l4n38 

Verbal-Pict. Translation 9.38 2.6$ 

Homonyms 7,21 2.29 

Jumbled Viords 23. 8U 2.69 

iiultiple Behavioral Group 2.60 I.I6 

Varied Emotional Rel. 9,61 2r22 

Selected ACL Scale 11.62 ^.iiii 

?Ton-Academic Accomplishment Scales 

Leadership 0.99 1.51^ 

Social Participation l.lLi 1.6$ 

Artistic lo6$ lo77 

Social Service 1.146 1.77 

Scientific 0.1$ Oo$U 

Business 0*70 1.01 

Humanistic/Cultural 1»88 l.$8 
Religious 1.36 .2.12 

llusical 0.29 0«71 

V/riting 0.96 1.03 

Social Service * 0.$0 0«8l 

Speech and Drama 0*$$ 1.12 

Total !o. of Items 11.7$ 7.96 

Improving Research Testing 

Fluency, Part I 7.00 . h.US 

Fluency, Part II $.81 3.22 

Workability 3.$7 2.66 

importance 1.98 O066 

Overall Rating (x 10) 27.0$ 

College Situations Problems 

Fluency 3.$3 1,86 

Flexibility 2.67 1.11 

Originality 3.21 

Structural Analysis 1.69 0..6$ 
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